## Loading required package: data.table
## data.table 1.14.8 using 6 threads (see ?getDTthreads). Latest news: r-datatable.com
## RStudio Community is a great place to get help: https://community.rstudio.com/c/tidyverse
## Loading required package: gridExtra
‘ipayipi’ was built for processing time-series data using ‘data.table’ functionality and speed for processing large datasets. This vignette introduces batch processing of time-series data using R ipayipi: covering extraction and standardisation of logger data. Specifically, the five steps outlined in this vignette are:
R ipayipi (install from GitHub),An overview of the code to execute these five steps are shown below. This code is all that is required to run each time new data is available. For more details and options for each step, see respective function documentation/help files.
## 1. load pacakge
library(ipayipi)
## 2. initiate pipeline ----
# define our general working directory: wherever data is going to be processed ...
wd <- "ipayipi/data-raw/eg_data/met_eg"
# define directory from which data will be sourced
sd <- "ipayipi/data-raw/eg_data/met_eg/dta_in"
# setup pipeline directory structure
pipe_house <- ipip_house(work_dir = wd, source_dir = sd)
## 3. read & imbibe in data ----
logger_data_import_batch(pipe_house)
imbibe_raw_batch(pipe_house, file_ext_in = ".dat",
data_setup = ipayipi::cs_toa5)
## 4. standardise data ----
header_sts(pipe_house)
phenomena_sts(pipe_house)
## store standardised data ----
# transfer standardised data to the nomvet_room
transfer_sts_files(pipe_house)
## 5. append standardised data files ----
append_station_batch(pipe_house)
## 6. identify gaps ----
gap_eval_batch(pipe_house)Organising time-series data from loggers, such as, weather stations
or ground-water sensors, requires no small amount of data structuring.
ipayipi helps make this process dynamic and structured, so
that, processing, down (and back up) a pipeline, is traceable.
Importantly, ipayipi preserves raw-data integrity and
initiates archival—so raw data is available in a standardised format—and
the processing of this data can be updated.
The package can be found on GitHub and can be
installed using the ‘devtools’ package:
devtools::install_github("SAEONData/ipayipi").
The ‘pipe_house’ is the name of the directory where we will initiate
a data-pipeline structure. To keep things simple, we will only
use our ‘pipe_house’ for defined data streams. In this vignette the data
stream consists of meteorological data gathered from a weather station
data logger. ipayipi handles most time-series data formats
readable from flat files into R. Data for this vignette can be
downloaded from GitHub in the package’s raw-data folder here.
# setting up the 'pipe_house'
## general pipeline working directory
wd <- "ipayipi/data-raw/eg_data/met_eg"
## data souce folder
sd <- "ipayipi/data-raw/eg_data/met_eg/dta_in"
## initiate pipeline
pipe_house <- ipip_house(work_dir = wd, source_dir = sd)
print(pipe_house)## $wait_room
## wait_room
## "ipayipi/data-raw/eg_data/met_eg/wait_room"
##
## $nomvet_room
## nomvet_room
## "ipayipi/data-raw/eg_data/met_eg/nomvet_room"
##
## $ipip_room
## ipip_room
## "ipayipi/data-raw/eg_data/met_eg/ipip_room"
##
## $raw_room
## raw_room
## "ipayipi/data-raw/eg_data/met_eg/raw_room"
##
## $source_dir
## [1] "ipayipi/data-raw/eg_data/met_eg/dta_in"
What has ipip_house() done? It has created the following
directories, if they don’t already exist*:
*NB! Running this function will not overwrite existing data.
In this step, data gets pulled from pipelines data source, that is,
the ‘source directory’ (pipe_house$source_room), into the
‘waiting room’ (pipe_house$wait_room). The example data
contains two-years of Cambell Scientific logger text files derived from
sensors on a SAEON meteorological
station in northern Maputaland, South Africa.
# copy data from source to the wait_room
logger_data_import_batch(pipe_house = pipe_house,
file_ext = ".dat", # the file extension (with period) of raw data files
verbose = FALSE, # set to TRUE to report progress in the terminal
unwanted = "02.2022" # excluding the import of Feb 2022 to make data 'gap'
)Now that some data is in the ‘wait_room’ directory we can read it
into R. Note the pre-set ‘data_setup’ option for Cambell Scientific TOA5
formatted files ipayipi::cs_toa5.
imbibe_raw_batch(pipe_house = pipe_house,
data_setup = ipayipi::cs_toa5, # standard for reading t0a5 formatted files
record_interval_type = "continuous"
)For more on data input formats, that is, the ‘data_setup’ argument,
see the help files of the imbibe_raw_logger_dt()
function (i.e., ?imbibe_raw_logger_dt).
Record-interval type is an important parameter. ipayipi
handles continuous, event-based
(discontinuous), and mixed time-series data types.
Record intervals get evaluated using the
record_interval_eval() function. Record interval
information will be important for further steps, such as, identifying
‘gaps’ or missing data automatically.
Both file-header information, plus other phenomena (variable) metadata, will now be standardised. The spelling/synonyms of file names and associated header metadata have to be scrutinised first. Only after header information gets standardised, can we move on to working on the phenomena. These steps are essential for automating file record appending and downstream data correction/processing (e.g., drift correction).
If it is the first-time running header_sts(), or new
synonyms get introduced into pipe-house directory,
header_sts() will produce a warning. This is because the
user needs to define new nomenclature standards. Unstandardised names (or columns) have
the preffix ‘uz’. These standards get stored in
a file called ‘nomtab.rns’ in the ‘waiting room’. If this file is
deleted—a new one will be generated—but the user will have to populate
the tables with synonym vocab.
The nomenclature table in the ‘waiting room’ can be updated from ‘csv’ format (or directly in R). If a new synonym gets introduced—the file containing new nomenclature will be skipped in further processing—a ‘csv’ version of the ‘nomtab.rns’ will be copied to the ‘waiting room’ for editing.
Only the following fields — with NAs —
require editing in the ‘nomtab’ ‘csv’:
record_interval_eval(). However, if there are in sufficient
data records, e.g., one record in a continuous data flow, this needs
manual checking.ipayipi::record_interval_eval().pt <- read.csv("ipayipi/data-raw/eg_data/met_eg/nomtab_display.csv")
kbl(pt) |>
kable_paper("hover") |>
kable_styling(font_size = 11) |>
column_spec(c(1, 7:8, 11), background = "#df9a86") |>
column_spec(c(2:6, 9:10, 12), background = "#aed8f0") |>
scroll_box(width = "100%", height = "400px")| uz_station | location | station | stnd_title | logger_type | logger_title | uz_record_interval_type | uz_record_interval | record_interval_type | record_interval | uz_table_name | table_name |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Mabasa AWS | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000 | mixed | 1_days | mixed | 1_days | TableDay | raw_1_days |
| Mabasa AWS | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000 | mixed | 1_hours | mixed | 1_hours | TableHour | raw_1_hours |
| Mabasa AWS | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000X | continuous | 1_days | continuous | 1_days | TableDay | raw_1_days |
| Mabasa AWS | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000X | continuous | 1_hours | continuous | 1_hours | TableHour | raw_1_hours |
| Station2 | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000X | continuous | 1_days | continuous | 1_days | TableDay | raw_1_days |
| Station2 | mcp | mabasa_aws | mcp_mabasa_aws | CR1000X | CR1000X | continuous | 1_hours | continuous | 1_hours | TableHour | raw_1_hours |
| Sibayi AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | CR200X | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Sibayi AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | CR200X | continuous | 5_mins | continuous | 5_mins | Five_Minutes | raw_5_mins |
| Sibayi AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | CR200X | continuous | 1_hours | continuous | 1_hours | Hourly_ | raw_1_hours |
| 84923 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| 84923 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| 84923 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| CR1000 - Vasi_Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| CR1000 - Vasi_Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| CR1000 - Vasi_Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| CR1000_Vasi Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| CR1000_Vasi Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | discnt | continuous | discnt | Daily | raw_1_days |
| CR1000_Vasi Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| CR1000_Vasi Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| Science Center AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Science Center AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| Science Center AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins |
| Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | mixed | 5_mins | mixed | 5_mins | Five_mins | raw_5_mins |
| Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | CR1000 | CR1000 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours |
| CR200Series | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| CR200Series | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 5_mins | continuous | 5_mins | Five_Minutes | raw_5_mins |
| CR200Series | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_hours | continuous | 1_hours | Hourly_ | raw_1_hours |
| Sibayi | mcp | sibayi_aws | mcp_sibayi_aws | CR300 | NA | continuous | 1_hours | continuous | 1_hours | Table1 | raw_1_hours |
| Sibayi | mcp | sibayi_aws | mcp_sibayi_aws | CR300 | NA | continuous | 1_days | continuous | 1_days | Table2 | raw_1_days |
| Sibayi | mcp | sibayi_aws | mcp_sibayi_aws | CR300 | NA | continuous | 5_mins | continuous | 5_mins | Table3 | raw_5_mins |
| Sibayi AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | discnt | continuous | discnt | Daily | raw_1_days |
| Sibayi Camp Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | mixed | 1_days | mixed | 1_days | Daily | raw_1_days |
| Sibayi Camp Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Sibayi Camp Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 5_mins | continuous | 5_mins | Five_Minutes | raw_5_mins |
| Sibayi Camp Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_hours | continuous | 1_hours | Hourly_ | raw_1_hours |
| Sibayi Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Sibayi Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 5_mins | continuous | 5_mins | Five_Minutes | raw_5_mins |
| Sibayi Mini AWS | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_hours | continuous | 1_hours | Hourly_ | raw_1_hours |
| Sibayi Mini Station | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_days | continuous | 1_days | Daily | raw_1_days |
| Sibayi Mini Station | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 5_mins | continuous | 5_mins | Five_Minutes | raw_5_mins |
| Sibayi Mini Station | mcp | sibayi_aws | mcp_sibayi_aws | CR200X | NA | continuous | 1_hours | continuous | 1_hours | Hourly_ | raw_1_hours |
Once NA values of the above fields have been populated
the edited ‘csv’ will be imbibed into the pipeline structure when
rerunning header_sts(pipe_house)—this function will imbibe
the most recently updated ‘csv’ nomenclature table from the ‘wait_room’
into the pipeline, and standardised header nomenclature.
In step with good tidy data standards, keep nomenclature to ‘snake case’ with no special characters (bar the useful underscore’). ’ Standardising phenomena metadata follows a similar process as for header-data standardisation. If the phenomena standards have been described and there is a ‘phentab.rps’ in the ‘waiting room’, running the below code updates all files phenomena details.
If there is no ‘phenomena table’ (‘phentab.rps’), one NA
values in the ‘csv’ copy need to be described. The following fields in
the ‘csv’ phentab must be populated:
Additional fields that are not mandatory include:
If an ‘f_convert’ factor (scroll right on the table below) is applied to phenomena, the standardised units must be different from the unstandardised units (uz_units) in the phenomena table. This ensures that phenomena that are appended have similar units.
| phen_name_full | phen_type | phen_name | units | measure | offset | var_type | uz_phen_name | uz_units | uz_measure | f_convert | sensor_id | notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Battery level: maximum voltage | Battery level | batt_max_v | v | max | 0 | num | BattV_Max | Volts | Max | NA | NA | NA |
| Battery level: maximum voltage | Battery level | batt_max_v | v | max | 0 | num | LoggerBatt_Max | Volts | Max | NA | NA | NA |
| Battery level: time of maximum | Battery level | batt_max_v_time | time | smp | 0 | posix | BattV_TMx | Volts | TMx | NA | NA | NA |
| Battery level: minimum voltage | Battery level | batt_min_v | v | min | 0 | num | BattV_Min | Volts | Min | NA | NA | NA |
| Battery level: time of minimum | Battery level | batt_min_v_time | time | smp | 0 | posix | BattV_TMn | Volts | TMn | NA | NA | NA |
| Data logger: serial number | Data logger: serial number | data_logger_sn | smp | smp | 0 | chr | DataloggerSerialNumber | Smp | NA | NA | NA | |
| Humidity, relative: maximum | Humidity, relative | humid_rel_max_pcnt | pcnt | max | 0 | num | RH_Max | % | Max | NA | NA | NA |
| Humidity, relative: time of maximum | Humidity, relative | humid_rel_max_time | time | smp | 0 | posix | RH_TMx | TMx | NA | NA | NA | |
| Humidity, relative: minimum | Humidity, relative | humid_rel_min | pcnt | min | 0 | num | RH_Min | % | Min | NA | NA | NA |
| Humidity, relative: percent | Humidity, relative | humid_rel_pcnt | pcnt | smp | 0 | num | RH | % | Smp | NA | NA | NA |
| Leaf wetness: total time con | Leaf wetness: time con | leaf_con_tot_time | mins | tot | 0 | num | LWMCon_Tot | Minutes | Tot | NA | NA | NA |
| Leaf wetness: total time dry | Leaf wetness: time dry | leaf_dry_tot_time | mins | tot | 0 | num | LWMDry_Tot | Minutes | Tot | NA | NA | NA |
| Leaf wetness: average | Leaf wetness: conductivity | leaf_wet_avg | mv | avg | 0 | num | LWmV_Avg | mV | Avg | NA | NA | NA |
| Leaf wetness: maximum | Leaf wetness | leaf_wet_max | mv | max | 0 | num | LWmV_Max | mV | Max | NA | NA | NA |
| Leaf wetness: time of maximum | Leaf wetness | leaf_wet_max_time | time | smp | 0 | posix | LWmV_TMx | TMx | NA | NA | NA | |
| Leaf wetness: minimum | Leaf wetness | leaf_wet_min | mv | min | 0 | num | LWmV_Min | mV | Min | NA | NA | NA |
| Leaf wetness: time of minimum | Leaf wetness | leaf_wet_min_time | time | smp | 0 | posix | LWmV_TMn | TMn | NA | NA | NA | |
| Leaf wetness: total time wet | Leaf wetness: time wet | leaf_wet_tot_time | mins | tot | 0 | num | LWMWet_Tot | Minutes | Tot | NA | NA | NA |
| Moisture, soil: average percent | Moisture, soil | moisture_soil_pcnt | pcnt | avg | 0 | num | VW_Avg | Avg | NA | NA | NA | |
| Pressure, atmosphere: sample | Pressure, atmosphere | pressure_atm | kpa | smp | 0 | num | BP_kPa | kPa | Smp | NA | NA | NA |
| Pressure, atmosphere: average | Pressure, atmosphere | pressure_atm_avg | kpa | avg | 0 | num | BPressure_Avg | kPa | Avg | NA | NA | NA |
| Pressure, atmosphere: average | Pressure, atmosphere | pressure_atm_avg | kpa | avg | 0 | num | BPressure_Avg | mbar | Avg | 0.1 | NA | Converted to kilopascals by multiplying by 0.1. |
| Pressure, atmosphere: time of maximum | Pressure, atmosphere | pressure_atm_max_time | time | smp | 0 | posix | BPressure_TMx | TMx | NA | NA | NA | |
| Pressure, atmosphere: minimum | Pressure, atmosphere | pressure_atm_min | kpa | min | 0 | num | BPressure_Min | kPa | Min | NA | NA | NA |
| Pressure, atmosphere: time of minimum | Pressure, atmosphere | pressure_atm_min_time | time | smp | 0 | posix | BPressure_TMn | TMn | NA | NA | NA | |
| Rainfall: total | Rainfall | rain_tot | mm | tot | 0 | num | Rain_Tot | mm | Tot | NA | NA | NA |
| Rainfall: total | Rainfall | rain_tot | mm | tot | 0 | num | Rain_mm_Tot | mm | Tot | NA | NA | NA |
| Solar radiation: average | Solar radiation | solar_rad_avg | mj_per_m2 | avg | 0 | num | SlrW_Avg | W/m^2 | Avg | NA | NA | NA |
| Solar radiation: maximum | Solar radiation | solar_rad_max | mj_per_m2 | max | 0 | num | SlrW_Max | W/m^2 | Max | NA | NA | NA |
| Solar radiation: time of maximum | Solar radiation | solar_rad_max_time | time | smp | 0 | posix | SlrW_TMx | TMx | NA | NA | NA | |
| Solar radiation: standard deviation | Solar radiation | solar_rad_sd | w_per_m2 | sd | 0 | num | SlrW_Std | W/m^2 | Std | NA | NA | NA |
| Solar radiation: total | Solar radiation | solar_rad_tot | mj_per_m2 | tot | 0 | num | SlrMJ_Tot | MJ/m^2 | Tot | NA | NA | NA |
| Temperature, air: average | Temperature, air | temp_air_avg | deg_c | avg | 0 | num | AirTC_Avg | Deg C | Avg | NA | NA | NA |
| Temperature, air: maximum | Temperature, air | temp_air_max | deg_c | max | 0 | num | AirTC_Max | Deg C | Max | NA | NA | NA |
| Temperature, air: time of maximum | Temperature, air | temp_air_max_time | time | smp | 0 | posix | AirTC_TMx | Deg C | TMx | NA | NA | NA |
| Temperature, air: minimum | Temperature, air | temp_air_min | deg_c | min | 0 | num | AirTC_Min | Deg C | Min | NA | NA | NA |
| Temperature, air: time of minimum | Temperature, air | temp_air_min_time | time | smp | 0 | posix | AirTC_TMn | Deg C | TMn | NA | NA | NA |
| Temperature, ground level: average | Temperature, ground level | temp_ground_avg | deg_c | avg | 0 | num | T107_C_Avg | Deg C | Avg | NA | NA | NA |
| Temperature, ground level: minimum | Temperature, ground level | temp_ground_min | deg_c | min | 0 | num | T107_C_Min | Deg C | Min | NA | NA | NA |
| Temperature, logger: average | Temperature, logger | temp_logg_avg | deg_c | avg | 0 | num | LoggerTemp_Avg | DegC | Avg | NA | NA | NA |
| Temperature, logger: maximum | Temperature, logger | temp_logg_max | deg_c | max | 0 | num | LoggerTemp_Max | DegC | Max | NA | NA | NA |
| Temperature, logger: time of maximum | Temperature, logger | temp_logg_max_time | time | smp | 0 | posix | LoggerTemp_TMx | TMx | NA | NA | NA | |
| Temperature, logger: minimum | Temperature, logger | temp_logg_min | deg_c | min | 0 | num | LoggerTemp_Min | DegC | Min | NA | NA | NA |
| Temperature, logger: time of minimum | Temperature, logger | temp_logg_min_time | time | smp | 0 | posix | LoggerTemp_TMn | TMn | NA | NA | NA | |
| Temperature, soil: average | Temperature, soil | temp_soil_avg | deg_c | avg | 0 | num | SoilTemp_Avg | Deg C | Avg | NA | NA | NA |
| Temperature, soil: maximum | Temperature, soil | temp_soil_max | deg_c | max | 0 | num | SoilTemp_Max | Deg C | Max | NA | NA | NA |
| Temperature, soil: time of maximum | Temperature, soil | temp_soil_max_time | time | smp | 0 | posix | SoilTemp_TMx | TMx | NA | NA | NA | |
| Temperature, soil: minimum | Temperature, soil | temp_soil_min | deg_c | min | 0 | num | SoilTemp_Min | Deg C | Min | NA | NA | NA |
| Temperature, soil: time of minimum | Temperature, soil | temp_soil_min_time | time | smp | 0 | posix | SoilTemp_TMn | TMn | NA | NA | NA | |
| UV radiation: average | UV radiation | uv_rad_avg | mj_per_m2 | avg | 0 | num | CUV5_W_Avg | W/m^2 | Avg | NA | NA | NA |
| UV radiation: average | UV radiation | uv_rad_avg | w_per_m2 | avg | 0 | num | UV_W_Avg | W/m^2 | Avg | NA | NA | NA |
| UV radiation: maximum | UV radiation | uv_rad_max | mj_per_m2 | max | 0 | num | CUV5_W_Max | W/m^2 | Max | NA | NA | NA |
| UV radiation: time of maximum | UV radiation | uv_rad_max_time | time | smp | 0 | posix | UV_W_TMx | TMx | NA | NA | NA | |
| UV radiation: standard deviation | UV radiation | uv_rad_sd | mj_per_m2 | sd | 0 | num | CUV5_W_Std | W/m^2 | Std | NA | NA | NA |
| UV radiation: sensitivity | UV radiation, sensor | uv_rad_sensi | desc | smp | 0 | num | UV_Sensitivity | Smp | NA | NA | NA | |
| UV radiation: sensor sn | UV radiation, sensor | uv_rad_sn | desc | smp | 0 | int | UV_SN | Smp | NA | NA | NA | |
| UV radiation: total | UV radiation | uv_rad_tot | mj_per_m2 | tot | 0 | num | CUV5_MJ_Tot | MJ/m^2 | Tot | NA | NA | NA |
| Wind direction: instantaneous | Wind direction | wind_dir | deg | smp | 0 | num | WindDir_D1_WVT | Deg | WVc | NA | NA | NA |
| Wind direction: standard deviation | Wind direction | wind_dir_sd | deg | sd | 0 | num | WindDir_SD1_WVT | Deg | WVc | NA | NA | NA |
| Wind speed: sample | Wind speed | wind_speed | m_per_sec | smp | 0 | num | WS_ms | meters/second | Smp | NA | NA | NA |
| Wind speed: average | Wind speed | wind_speed_avg | m_per_sec | avg | 0 | num | WS_ms_Avg | meters/second | Avg | NA | NA | NA |
| Wind speed: maximum | Wind speed | wind_speed_max | m_per_sec | max | 0 | num | WS_ms_Max | meters/sec | Max | NA | NA | NA |
| Wind speed: time of maximum | Wind speed | wind_speed_max_time | time | smp | 0 | posix | WS_ms_TMx | meters/second | TMx | NA | NA | NA |
| Wind speed: minimum | Wind speed | wind_speed_min | m_per_sec | min | 0 | num | WS_ms_Min | meters/second | Min | NA | NA | NA |
| Wind speed: minimum | Wind speed | wind_speed_min | m_per_sec | min | 0 | num | WSpeed_Min | meters/second | Min | NA | NA | NA |
| Wind speed: time of minimum | Wind speed | wind_speed_min_time | time | smp | 0 | posix | WS_ms_TMn | meters/second | TMn | NA | NA | NA |
After filling in details, to replace NA values, rerun
phenomena_sts(pipe_house), to imbibe the updated phenomena
descriptions, and update the logger data being standardised.
Standardised data files get transferred to the ‘nomenclature vetted’ directory (‘nomtab room’) using the function below. After being transferred, files in the waiting room (except the nomtab and phentab standards) are automatically removed.
Archiving raw data files: Before removing raw
unstandardised files—if there is a ‘raw_room’ directory in the pipeline
working directory—raw input data files will be copied to this directory
and filed in folders by year and month of the lasted date of recording.
This is done by the imbibe_raw_batch() function.
The append_station_batch() function updates station
files in the ‘ipip_room’ with files from the ‘nomvet_room’.
# append station files + metadata records
# note the 'cores' argument --- parallel processing supported on Linux systems
append_station_batch(pipe_house, cores = 2)Now that a station file has been generated for the Vasi Science Centre weather station we can check what tables have been created/appended. Station files are maintained in the ‘ipip_room’ of the pipeline’s folder structure.
# list station files in the ipip directory
sf <- dta_list(
input_dir = pipe_house$ipip_room, # search directory
file_ext = ".ipip", # note the station's default file extension
)
# check what stations are in the ipip room
print(sf)## [1] "mcp_vasi_science_centre_aws.ipip"
# read in the station file
sf <- readRDS(file.path(pipe_house$ipip_room, sf[1]))
# names of the tables stored in the station file
names(sf)## [1] "data_summary" "gaps" "logg_interfere" "phen_data_summary" "phens" "raw_1_days" "raw_1_hours" "raw_5_mins"
Our station file has three ‘raw’ data tables, with 5 minute, daily, and monthly data.
## id date_time wind_speed wind_dir wind_dir_sd temp_air_avg humid_rel_pcnt solar_rad_avg uv_rad_avg rain_tot temp_ground_avg
## 1: 0 2020-06-11 12:35:00 3.29 309.2 42.72 31.27 38.04 584.1 24.86 0 32.92
# use kableExtra to view first 20 data rows --- default printing of tables doesn't
# looks good in html
kbl(sf$raw_5_min[1:20, ]) |>
kable_paper("hover") |>
kable_styling(font_size = 11) |>
scroll_box(width = "100%", height = "400px")| id | date_time | wind_speed | wind_dir | wind_dir_sd | temp_air_avg | humid_rel_pcnt | solar_rad_avg | uv_rad_avg | rain_tot | temp_ground_avg |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020-06-11 12:35:00 | 3.290 | 309.2 | 42.72 | 31.27 | 38.04 | 584.1 | 24.86 | 0 | 32.92 |
| 1 | 2020-06-11 12:40:00 | 3.722 | 338.0 | 27.10 | 30.89 | 38.86 | 582.1 | 24.67 | 0 | 32.69 |
| 2 | 2020-06-11 12:45:00 | 3.897 | 322.8 | 19.89 | 30.95 | 39.30 | 578.1 | 24.41 | 0 | 32.58 |
| 3 | 2020-06-11 12:50:00 | 3.545 | 340.0 | 28.47 | 30.91 | 38.91 | 572.1 | 24.00 | 0 | 32.45 |
| 4 | 2020-06-11 12:55:00 | 4.050 | 333.9 | 25.85 | 30.92 | 39.44 | 566.3 | 23.60 | 0 | 32.47 |
| 5 | 2020-06-11 13:00:00 | 3.847 | 344.2 | 31.73 | 30.89 | 38.24 | 556.8 | 22.95 | 0 | 32.55 |
| 6 | 2020-06-11 13:05:00 | 3.442 | 334.3 | 32.61 | 30.96 | 37.96 | 553.2 | 22.87 | 0 | 32.63 |
| 7 | 2020-06-11 13:10:00 | 3.085 | 332.0 | 28.55 | 31.17 | 37.76 | 545.6 | 22.51 | 0 | 32.63 |
| 8 | 2020-06-11 13:15:00 | 3.772 | 326.0 | 37.08 | 31.31 | 37.47 | 539.7 | 22.05 | 0 | 32.83 |
| 9 | 2020-06-11 13:20:00 | 3.795 | 343.7 | 32.37 | 31.07 | 37.19 | 528.5 | 21.49 | 0 | 32.74 |
| 10 | 2020-06-11 13:25:00 | 3.403 | 337.0 | 28.52 | 31.22 | 36.62 | 521.3 | 21.38 | 0 | 32.70 |
| 11 | 2020-06-11 13:30:00 | 3.165 | 337.8 | 38.35 | 31.41 | 35.68 | 516.0 | 21.28 | 0 | 32.81 |
| 12 | 2020-06-11 13:35:00 | 3.307 | 334.6 | 32.02 | 31.51 | 36.00 | 507.3 | 20.91 | 0 | 32.97 |
| 13 | 2020-06-11 13:45:00 | 3.006 | 348.9 | 21.52 | 31.45 | 35.30 | 491.1 | 20.20 | 0 | 32.98 |
| 14 | 2020-06-11 13:55:00 | 2.538 | 326.6 | 0.00 | 31.43 | 34.45 | 473.1 | 19.42 | 0 | 32.74 |
| 15 | 2020-06-11 14:00:00 | 3.470 | 331.2 | 27.09 | 31.58 | 34.13 | 467.7 | 19.18 | 0 | 32.77 |
| 16 | 2020-06-11 14:05:00 | 3.690 | 326.4 | 36.23 | 31.61 | 34.41 | 454.5 | 18.53 | 0 | 32.76 |
| 17 | 2020-06-11 14:10:00 | 2.800 | 339.9 | 24.61 | 32.08 | 33.83 | 429.4 | 17.54 | 0 | 33.21 |
| 18 | 2020-06-11 14:15:00 | 3.100 | 341.2 | 40.78 | 31.85 | 34.96 | 422.1 | 17.27 | 0 | 33.06 |
| 19 | 2020-06-11 14:20:00 | 3.613 | 332.9 | 32.83 | 31.67 | 34.00 | 411.2 | 16.69 | 0 | 32.80 |
This table contains summary information on the origin of each data file used to make up the station file.
# using kableExtra
kbl(sf$data_summary) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
scroll_box(width = "100%", height = "400px")| dsid | file_format | uz_station | location | station | stnd_title | start_dttm | end_dttm | logger_type | logger_title | logger_sn | logger_os | logger_program_name | logger_program_sig | uz_record_interval_type | uz_record_interval | record_interval_type | record_interval | uz_table_name | table_name | nomvet_name | file_origin |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-06-12 00:00:00 | 2021-02-10 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20200612-20210210__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210210__1.dat |
| 2 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-12-09 00:00:00 | 2021-01-12 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20201209-20210112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily12.01.2021__1.dat |
| 3 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-01-13 00:00:00 | 2021-04-14 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210113-20210414__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily14.04.2021__1.dat |
| 4 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-02-11 00:00:00 | 2021-03-09 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210211-20210309__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210309__1.dat |
| 5 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-03-10 00:00:00 | 2021-05-12 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210310-20210512__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210512__1.dat |
| 6 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-13 00:00:00 | 2021-06-16 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210513-20210616__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily16.06.2021__1.dat |
| 7 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-13 00:00:00 | 2021-07-21 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210513-20210721__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210721__1.dat |
| 8 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-06-17 00:00:00 | 2021-08-11 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210617-20210811__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily11.08.2021__1.dat |
| 9 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-08-12 00:00:00 | 2021-09-13 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210812-20210913__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily13.09.2021__1.dat |
| 10 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-09-14 00:00:00 | 2021-10-13 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20210914-20211013__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily13.10.2021__1.dat |
| 11 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-10-14 00:00:00 | 2021-11-08 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20211014-20211108__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily16.06.2021__2.dat |
| 12 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-11-09 00:00:00 | 2021-12-07 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20211109-20211207__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__1.dat |
| 13 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-12-08 00:00:00 | 2022-01-12 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20211208-20220112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__2.dat |
| 14 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-02-10 00:00:00 | 2022-03-07 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220210-20220307__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.03.2022__1.dat |
| 15 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-03-08 00:00:00 | 2022-04-12 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220308-20220412__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220412__1.dat |
| 16 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-04-13 00:00:00 | 2022-05-11 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220413-20220511__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220511__1.dat |
| 17 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-05-12 00:00:00 | 2022-06-13 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220512-20220613__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220613__1.dat |
| 18 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-06-14 00:00:00 | 2022-07-12 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220614-20220712__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__3.dat |
| 19 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-07-13 00:00:00 | 2022-08-10 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220713-20220810__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__4.dat |
| 20 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-08-11 00:00:00 | 2022-09-15 00:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_days | continuous | 1_days | Daily | raw_1_days | mcp_vasi_science_centre_aws_1_days_20220811-20220915__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__5.dat |
| 21 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-06-11 13:00:00 | 2021-02-10 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20200611-20210210__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210210__1.dat |
| 22 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-12-08 15:00:00 | 2021-01-12 11:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20201208-20210112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly12.01.2021__1.dat |
| 23 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-01-12 12:00:00 | 2021-04-14 13:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210112-20210414__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__1.dat |
| 24 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-02-10 15:00:00 | 2021-03-09 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210210-20210309__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210309__1.dat |
| 25 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-03-09 15:00:00 | 2021-05-12 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210309-20210512__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210512__1.dat |
| 27 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-12 15:00:00 | 2021-07-21 13:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210512-20210721__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210721__1.dat |
| 26 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-12 16:00:00 | 2021-06-16 11:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210512-20210616__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly16.06.2021__1.dat |
| 28 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-06-16 12:00:00 | 2021-08-11 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210616-20210811__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly11.08.2021__1.dat |
| 29 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-08-11 16:00:00 | 2021-09-13 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210811-20210913__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly13.09.2021__1.dat |
| 30 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-09-13 16:00:00 | 2021-10-13 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20210913-20211013__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly13.10.2021__1.dat |
| 31 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-10-13 16:00:00 | 2021-11-08 12:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20211013-20211108__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__2.dat |
| 32 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-11-08 13:00:00 | 2021-12-07 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20211108-20211207__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__3.dat |
| 33 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-12-07 15:00:00 | 2022-01-12 11:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20211207-20220112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__4.dat |
| 34 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-02-09 17:00:00 | 2022-03-07 16:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220209-20220307__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly07.03.2022__1.dat |
| 35 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-03-07 17:00:00 | 2022-04-12 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220307-20220412__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220412__1.dat |
| 36 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-04-12 15:00:00 | 2022-05-11 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220412-20220511__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220511__1.dat |
| 37 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-05-11 15:00:00 | 2022-06-13 13:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220511-20220613__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220613__1.dat |
| 38 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-06-13 14:00:00 | 2022-07-12 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220613-20220712__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__5.dat |
| 39 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-07-12 16:00:00 | 2022-08-10 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220712-20220810__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__6.dat |
| 40 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-08-10 16:00:00 | 2022-09-15 11:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 1_hours | continuous | 1_hours | Hourly | raw_1_hours | mcp_vasi_science_centre_aws_1_hours_20220810-20220915__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__7.dat |
| 41 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-06-11 12:35:00 | 2021-02-10 14:45:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20200611-20210210__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210210__1.dat |
| 42 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2020-12-08 14:55:00 | 2021-01-12 11:35:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20201208-20210112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins12.01.2021__1.dat |
| 43 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-01-12 11:40:00 | 2021-04-14 13:40:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210112-20210414__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__1.dat |
| 44 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-02-10 14:50:00 | 2021-03-09 14:30:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210210-20210309__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210309__1.dat |
| 45 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-03-09 14:35:00 | 2021-05-12 14:55:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210309-20210512__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210512__1.dat |
| 47 | TOA5 | 88296 | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-12 15:00:00 | 2021-07-21 13:45:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210512-20210721__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210721__1.dat |
| 46 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-05-12 15:20:00 | 2021-06-16 11:50:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210512-20210616__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins16.06.2021__1.dat |
| 48 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-06-16 11:55:00 | 2021-08-11 15:20:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210616-20210811__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins11.08.2021__1.dat |
| 49 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-08-11 15:25:00 | 2021-09-13 15:45:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210811-20210913__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins13.09.2021__1.dat |
| 50 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-09-13 15:50:00 | 2021-10-13 15:40:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20210913-20211013__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins13.10.2021__1.dat |
| 51 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-10-13 15:45:00 | 2021-11-08 12:55:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20211013-20211108__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__2.dat |
| 52 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-11-08 13:00:00 | 2021-12-07 14:40:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20211108-20211207__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__3.dat |
| 53 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2021-12-07 14:45:00 | 2022-01-12 11:55:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20211207-20220112__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__4.dat |
| 54 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-02-09 16:10:00 | 2022-03-07 16:10:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220209-20220307__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins07.03.2022__1.dat |
| 55 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-03-07 16:15:00 | 2022-04-12 14:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220307-20220412__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220412__1.dat |
| 56 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-04-12 14:05:00 | 2022-05-11 14:35:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220412-20220511__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220511__1.dat |
| 57 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-05-11 14:40:00 | 2022-06-13 13:35:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220511-20220613__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220613__1.dat |
| 58 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-06-13 13:40:00 | 2022-07-12 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220613-20220712__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__5.dat |
| 59 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-07-12 15:05:00 | 2022-08-10 15:00:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220712-20220810__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__6.dat |
| 60 | TOA5 | Science Centre AWS | mcp | vasi_science_centre_aws | mcp_vasi_science_centre_aws | 2022-08-10 15:05:00 | 2022-09-15 11:45:00 | CR1000 | NA | 88296 | CR1000.Std.31.08 | CPU:Vasi_Science Centre AWS.CR1 | 64103 | continuous | 5_mins | continuous | 5_mins | Five_mins | raw_5_mins | mcp_vasi_science_centre_aws_5_mins_20220810-20220915__1.ipi | ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__7.dat |
A station file version of phenomena standards. Note each phenomena variation/synonym has a unique identifier (‘phid’) within the scope of this station.
# using kableExtra
kbl(sf$phens) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
scroll_box(width = "100%", height = "400px")| phid | phen_name_full | phen_type | phen_name | units | measure | offset | var_type | uz_phen_name | uz_units | uz_measure | f_convert | sensor_id | notes | table_name | origin |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Humidity, relative: maximum | Humidity, relative | humid_rel_max_pcnt | pcnt | max | 0 | num | RH_Max | % | Max | NA | NA | NA | raw_1_days | s_f |
| 2 | Humidity, relative: minimum | Humidity, relative | humid_rel_min | pcnt | min | 0 | num | RH_Min | % | Min | NA | NA | NA | raw_1_days | s_f |
| 3 | Rainfall: total | Rainfall | rain_tot | mm | tot | 0 | num | Rain_mm_Tot | mm | Tot | NA | NA | NA | raw_1_days | s_f |
| 4 | Solar radiation: maximum | Solar radiation | solar_rad_max | w_per_m2 | max | 0 | num | SlrW_Max | W/m^2 | Max | NA | NA | NA | raw_1_days | s_f |
| 5 | Solar radiation: standard deviation | Solar radiation | solar_rad_sd | w_per_m2 | sd | 0 | num | SlrW_Std | W/m^2 | Std | NA | NA | NA | raw_1_days | s_f |
| 6 | Temperature, air: maximum | Temperature, air | temp_air_max | deg_c | max | 0 | num | AirTC_Max | Deg C | Max | NA | NA | NA | raw_1_days | s_f |
| 7 | Temperature, air: minimum | Temperature, air | temp_air_min | deg_c | min | 0 | num | AirTC_Min | Deg C | Min | NA | NA | NA | raw_1_days | s_f |
| 8 | Temperature, ground level: average | Temperature, ground level | temp_ground_avg | deg_c | avg | 0 | num | T107_C_Avg | Deg C | Avg | NA | NA | NA | raw_1_days | s_f |
| 9 | Temperature, ground level: minimum | Temperature, ground level | temp_ground_min | deg_c | min | 0 | num | T107_C_Min | Deg C | Min | NA | NA | NA | raw_1_days | s_f |
| 10 | UV radiation: maximum | UV radiation | uv_rad_max | w_per_m2 | max | 0 | num | CUV5_W_Max | W/m^2 | Max | NA | NA | NA | raw_1_days | s_f |
| 11 | UV radiation: standard deviation | UV radiation | uv_rad_sd | w_per_m2 | sd | 0 | num | CUV5_W_Std | W/m^2 | Std | NA | NA | NA | raw_1_days | s_f |
| 12 | UV radiation: total | UV radiation | uv_rad_tot | mj_per_m2 | tot | 0 | num | CUV5_MJ_Tot | MJ/m^2 | Tot | NA | NA | NA | raw_1_days | s_f |
| 13 | Wind direction: sample | Wind direction | wind_dir | deg | smp | 0 | num | WindDir_D1_WVT | Deg | WVc | NA | NA | NA | raw_1_days | s_f |
| 14 | Wind direction: standard deviation | Wind direction | wind_dir_sd | deg | sd | 0 | num | WindDir_SD1_WVT | Deg | WVc | NA | NA | NA | raw_1_days | s_f |
| 15 | Wind speed: average | Wind speed | wind_speed_avg | m_per_sec | avg | 0 | num | VW_Avg | Avg | NA | NA | NA | raw_1_days | s_f | |
| 16 | Wind speed: maximum | Wind speed | wind_speed_max | m_per_sec | max | 0 | num | WS_ms_Max | meters/second | Max | NA | NA | NA | raw_1_days | s_f |
| 17 | Wind speed: sample | Wind speed | wind_speed | m_per_sec | smp | 0 | num | WS_ms_S_WVT | meters/second | WVc | NA | NA | NA | raw_1_days | s_f |
| 18 | Battery level: minimum voltage | Battery level | batt_min_v | v | min | 0 | num | BattV_Min | Volts | Min | NA | NA | NA | raw_1_hours | s_f |
| 19 | Humidity, relative: percent | Humidity, relative | humid_rel_pcnt | pcnt | smp | 0 | num | RH | % | Smp | NA | NA | NA | raw_1_hours | s_f |
| 20 | Pressure, atmosphere: sample | Pressure, atmosphere | pressure_atm | kpa | smp | 0 | num | BP_kPa | kPa | Smp | NA | NA | NA | raw_1_hours | s_f |
| 3 | Rainfall: total | Rainfall | rain_tot | mm | tot | 0 | num | Rain_mm_Tot | mm | Tot | NA | NA | NA | raw_1_hours | s_f |
| 22 | Solar radiation: average | Solar radiation | solar_rad_avg | w_per_m2 | avg | 0 | num | SlrW_Avg | W/m^2 | Avg | NA | NA | NA | raw_1_hours | s_f |
| 23 | Temperature, air: average | Temperature, air | temp_air_avg | deg_c | avg | 0 | num | AirTC_Avg | Deg C | Avg | NA | NA | NA | raw_1_hours | s_f |
| 8 | Temperature, ground level: average | Temperature, ground level | temp_ground_avg | deg_c | avg | 0 | num | T107_C_Avg | Deg C | Avg | NA | NA | NA | raw_1_hours | s_f |
| 9 | Temperature, ground level: minimum | Temperature, ground level | temp_ground_min | deg_c | min | 0 | num | T107_C_Min | Deg C | Min | NA | NA | NA | raw_1_hours | s_f |
| 26 | UV radiation: average | UV radiation | uv_rad_avg | w_per_m2 | avg | 0 | num | CUV5_W_Avg | W/m^2 | Avg | NA | NA | NA | raw_1_hours | s_f |
| 13 | Wind direction: sample | Wind direction | wind_dir | deg | smp | 0 | num | WindDir_D1_WVT | Deg | WVc | NA | NA | NA | raw_1_hours | s_f |
| 14 | Wind direction: standard deviation | Wind direction | wind_dir_sd | deg | sd | 0 | num | WindDir_SD1_WVT | Deg | WVc | NA | NA | NA | raw_1_hours | s_f |
| 15 | Wind speed: average | Wind speed | wind_speed_avg | m_per_sec | avg | 0 | num | VW_Avg | Avg | NA | NA | NA | raw_1_hours | s_f | |
| 17 | Wind speed: sample | Wind speed | wind_speed | m_per_sec | smp | 0 | num | WS_ms_S_WVT | meters/second | WVc | NA | NA | NA | raw_1_hours | s_f |
| 19 | Humidity, relative: percent | Humidity, relative | humid_rel_pcnt | pcnt | smp | 0 | num | RH | % | Smp | NA | NA | NA | raw_5_mins | NA |
| 3 | Rainfall: total | Rainfall | rain_tot | mm | tot | 0 | num | Rain_mm_Tot | mm | Tot | NA | NA | NA | raw_5_mins | NA |
| 22 | Solar radiation: average | Solar radiation | solar_rad_avg | w_per_m2 | avg | 0 | num | SlrW_Avg | W/m^2 | Avg | NA | NA | NA | raw_5_mins | NA |
| 23 | Temperature, air: average | Temperature, air | temp_air_avg | deg_c | avg | 0 | num | AirTC_Avg | Deg C | Avg | NA | NA | NA | raw_5_mins | NA |
| 8 | Temperature, ground level: average | Temperature, ground level | temp_ground_avg | deg_c | avg | 0 | num | T107_C_Avg | Deg C | Avg | NA | NA | NA | raw_5_mins | NA |
| 26 | UV radiation: average | UV radiation | uv_rad_avg | w_per_m2 | avg | 0 | num | CUV5_W_Avg | W/m^2 | Avg | NA | NA | NA | raw_5_mins | NA |
| 13 | Wind direction: sample | Wind direction | wind_dir | deg | smp | 0 | num | WindDir_D1_WVT | Deg | WVc | NA | NA | NA | raw_5_mins | NA |
| 14 | Wind direction: standard deviation | Wind direction | wind_dir_sd | deg | sd | 0 | num | WindDir_SD1_WVT | Deg | WVc | NA | NA | NA | raw_5_mins | NA |
| 17 | Wind speed: sample | Wind speed | wind_speed | m_per_sec | smp | 0 | num | WS_ms_S_WVT | meters/second | WVc | NA | NA | NA | raw_5_mins | NA |
Note the ‘phid’ link in the temporal phenomena summary below.
# using kableExtra
kbl(sf$phen_data_summary) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
scroll_box(width = "100%", height = "400px")| phid | phen_name | start_dttm | end_dttm | table_name |
|---|---|---|---|---|
| 1 | humid_rel_max_pcnt | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 2 | humid_rel_min | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 3 | rain_tot | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 4 | solar_rad_max | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 5 | solar_rad_sd | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 6 | temp_air_max | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 7 | temp_air_min | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 8 | temp_ground_avg | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 9 | temp_ground_min | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 10 | uv_rad_max | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 11 | uv_rad_sd | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 12 | uv_rad_tot | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 13 | wind_dir | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 14 | wind_dir_sd | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 17 | wind_speed | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 15 | wind_speed_avg | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 16 | wind_speed_max | 2020-06-12 00:00:00 | 2022-09-15 00:00:00 | raw_1_days |
| 18 | batt_min_v | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 19 | humid_rel_pcnt | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 20 | pressure_atm | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 3 | rain_tot | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 22 | solar_rad_avg | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 23 | temp_air_avg | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 8 | temp_ground_avg | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 9 | temp_ground_min | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 26 | uv_rad_avg | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 13 | wind_dir | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 14 | wind_dir_sd | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 17 | wind_speed | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 15 | wind_speed_avg | 2020-06-11 13:00:00 | 2022-09-15 11:00:00 | raw_1_hours |
| 19 | humid_rel_pcnt | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 3 | rain_tot | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 22 | solar_rad_avg | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 23 | temp_air_avg | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 8 | temp_ground_avg | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 26 | uv_rad_avg | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 13 | wind_dir | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 14 | wind_dir_sd | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
| 17 | wind_speed | 2020-06-11 12:35:00 | 2022-09-15 11:45:00 | raw_5_mins |
Phenomena append notes: When appending phenomena tables, if there is overlapping data, ‘ipayipi’ will examine each overlapping phenomena series in turn and overwrite either new data (e.g., additions to a station file) or the station file data. Missing data (NAs) will however not be overwritten. The phenomena data summary keeps a temporal record of how phenomena have been appended. Maintaining these records helps data processing down the pipeline.
Checking for data ‘gaps’ in continuous data streams can be fairly
straight forward—just highlight the missing/NA values. But with
discontinuous or event based data things are more nuanced.
gap_eval_batch() identifies gap periods, specifically where
a logger was not recording, in continuous and discontinuous time-series
data. Here the data is continuous so identifying gaps is simple.
gap_eval_batch(pipe_house, cores = 3)
# read in the station file
sf <- readRDS(file.path("ipayipi/data-raw/eg_data/met_eg",
"ipip_room/mcp_vasi_science_centre_aws.ipip"))
sf$gaps## gid euid gap_type phen table_name gap_start gap_end dt_diff_s gap_problem_thresh_s problem_gap notes
## 1: 13 NA auto logger raw_1_days 2022-01-12 00:00:00 2022-02-10 00:00:00 2505600 secs 86400 TRUE <NA>
## 2: 13 NA auto logger raw_1_hours 2022-01-12 11:00:00 2022-02-09 17:00:00 2440800 secs 3600 TRUE <NA>
## 3: 13 NA auto logger raw_5_mins 2022-01-12 11:55:00 2022-02-09 16:10:00 2434500 secs 300 TRUE <NA>
Note the table and graph correctly show that February 2022 is missing in all raw data tables—we omitted importing this data using the ‘unwanted’ parameter in the import data stage above. Hover over the graph—interact.
p <- plot_raw_availability(pipe_house)
p <- p$plot_availability + scale_colour_sunset(discrete = TRUE) +
labs(color = "Station") +
theme(legend.position = "none")
plotly::ggplotly(p)Gaps highlighted in the dark-red colour.